Deep multimodal fusion for semantic image segmentation: A survey

Recent advances in deep learning have shown excellent performance various scene understanding tasks. However, some complex environments or under challenging conditions, it is necessary to employ multiple modalities that provide complementary information on the same scene. A variety of studies demonstrated multimodal fusion for semantic image segmentation achieves significant improvement. These approaches take benefits sources and generate an optimal joint prediction automatically. This paper describes essential background concepts relevant applications computer vision. In particular, we a systematic survey methodologies, datasets, quantitative evaluations benchmark datasets. Existing methods are summarized according common taxonomy: early fusion, late hybrid fusion. Based their performance, analyze strengths weaknesses different strategies. Current challenges design choices discussed, aiming reader with comprehensive heuristic view segmentation.